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Section 1 



INTRODUCTION 



Summary 

The utility of computer-based online information retrieval with the ERIC Research 
In Education files was tested by members of the EMC Clearinghouse at Stanford 
University and by the Regional Office of Education at San Francisco. Using a 
display-oriented retrieval system called DIALOG, which was developed by the 
Information Sciences organization of the Lockheed Palo Alto Research Laboratory, 
EMC personnel conducted more than 800 searches and printed out more than 
51,000 citations and abstracts over a 1-year period. 

Evaluations of the system were conducted at both locations. Users were most 
impressed by the speed with which answers to search questions could be obtained, 
and by the way visual interaction with the file via the display terminal sigmficantly 
extended their search horizons. Nearly all users (90%) were satisfied with search 
results. Particularly useful system features were felt to be the display and the 
typewriter scratch-pad record that accompanies the search process. 

The system was used principally for retrospective retrieval and for generation of 
special bibliographies, au example of which is included in Section 4. Estimates of 
potential demand for use of the system range from 10, 000 to 125, 000 requests per 
year. Examples of additional uses of the system include current-av/areness announce- 
ments and evaluation of requests for grants. 

While not readily measurable in a quantitative sense, the principal benefit that the 
system provides is the capability to quickly access reported information so that 
the researcher can effectively accumulate up-to-date knowledge within his 
profession. 

Introduction 



The information explosion of the past decade has been accompanied by a prolif- 
eration of techniques to assist in information retrieval. Most familiar are the 
published catalogs of bibliographies and descriptive abstracts frequently 
organized by subject and cross-referenced by author, contract number, cor- 
porate source, etc. These catalogs are widely disseminated and serve both 
announcement and retrieval source roles. 

Equally interesting has been the development of computer-oriented information 
retrieval systems. Recently it has become possible to store the catalog in a 
central computer and provide access to the computer using television-like key- 
board display devices, which, in many cases, are connected to the computer 
via ordinary telephone lines. The user is provided a set of command keys on 
his terminal keyboard with which he can instruct the computer to search for 
abstracts containing any C' sired combination of characteristics. Having reviewed 
the results of one query, he may then modify his question to further refine the 
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query or to branch off onto some interesting related topic . At any time during the 
course of the search session, the user may print out particular items of interest 
or even complete bibliographies. 

In the subject contract, Office of Education personnel were provided with use of a 
state-of-the-art information retrieval language that has been applied to Research 
in Education and Historical Reports data bases. The language, known as DIALOG, * 
was developed by the Information Sciences group of the Lockheed Palo Alto 
Research Laboratory. (See Appendix A for a reprint of the Users Manual, which 
describes the system in detail.) Remote terminals were installed at the ERIC 
Clearinghouse on Educational Media and Technology at Stanford University, at the 
Office of Education Regional Office in San Francisco, and at ERIC Headquarters 
in Washington, D. C. Office of Education personnel and others had conducted 
interactive searches on the ERIC data bases for a period of just over a year at the 
time this report was prepared. 

A description of the experiment, analysis of the results, and conclusions in terms of 
potential demand and expected benefits are presented in this report. The remainder 
of this section outlines the work performed and the methods used. Section 2 dis- 
cusses the findings and analysis. Section 3 presents conclusions, and Section 4 con- 
tains cited references and a bibliography. The appendixes contain the ERIC/DIALOG 
Users Manual, examples of invitation letters sent out from centers, and a detailed 
summary of search results. 

Methods 

The work performed in this research included loading ERIC files onto the Lockheed 
computer; installation of remote terminals at Stanford, San Francisco, and 
Washington, D. C. ; and provision of a computer retrieval service to those terminals 
for scheduled operating periods. In addition, an evaluation of a computer- 
controlled microfilm display system was conducted, and a continuous 3-day presen-' 
tation at the annual meeting of the California Association of Secondary School 
Administrators (CASSA) was given. 

In early 1969, computer magnetic tapes containing 12,300 bibliographies and des- 
criptive abstracts from Research in Education and Historical Reports files were 
loaded onto random-access computer storage devices at the Lockheed computer 
center. Index and cross-reference files were created from fields in the master 
records based on discussion with ERIC personnel (see Appendix A, pp. 13 — 14, 
for examples of input records and field indexing). 

A remote terminal consisting of an IBM 2260 display and a 1053 printer (see 
Appendix A, p. 2) was installed at the ERIC Clearinghouse on Educational Media 
and Technology at Stanford University. This terminal was coupled to the Lockheed 
computer via a Bell 201 data set with a transmission rate of 240 characters per 
second. Clearinghouse personnel conducted interactive searches using the termi- 
nal in response to mail and telephone requests from the education community. In 



*A later version developed for NASA is known as RECON, an acronym for Remote 
Console. 



addition, an invitation (Appendix B) was mailed to local educators to enable them 
to become familiar with the service. Based on their experience with the terminal, 
Stanford published a report evaluating the system operation (Ref. 1), which is 
discussed in Section 2. 

In third quarter of 1969, the file was updated through August 1969. File size at 
this time included approximately 20, 000 titles. A thesaurus file was added through 
which users could obtain conceptually related index terms on the display terminal. 

A later version of DIALOG was installed that provided some additional conveniences 
to users. A second remote terminal (CC30, manufactured by Computer Communi- 
cations, Inc.) was installed at the Office of Education Regional Office in San 
Francisco. This terminal, although functionally similar to that installed at Stanford, 
was considerably more economical in both lease and operating costs and in the 
physical size of support equipment. ♦ This terminal utilizes a Bell 202 drta set with 
a transmission rate of 120 characters per second. 

In parallel with the service operations at Stanford and San Francisco, an interesting 
experiment was conducted with a small microfilm company, Microform Data 
Systems, Inc. A sample of 2, 000 of the ERIC abstracts was reduced to microfilm 
and placed on a computer-dxiven M.D. S. viewer. In this environment, the user 
could conduct a search of the indexes residing in the computer to identify candidate 
abstracts. Any abstract could then be displayed on the viewer with a simple 
command to the computer. This configuration was demonstrated at the 1969 Fall 
Joint Computer Conference in Las Vegas. 

In late 1969, a final terminal, identical to that in San Francisco, was installed at 
ERIC headquarters in Washington, D. C. This terminal is part of an Educational 
Information Center project. 

In April 1969, the San Francisco terminal w^s temporarily relocated to San Diego 
for the annual Conference of the California Association of Secondary School 
Administrators. During the conference a large number of searches were conducted 
by attendees with results mailed directly to their homes to await their return. 

A sample listing of search titles is provided in Appendix C to illustrate the variety 
of topics of current interest within education and to demonstrate the practical 
operational nature of online retrieval methods. 

Summary of Usage Statistics 

Usage statistics are summarized in this section for Stanford, San Francisco, and 
San Diego (California Association of Secondary School Administrators Conference). 
Detailed frequency distributions appear in Appendix C. Characteristics measured 
include elapsed time per search session (Table C-1), number of online references 
to the thesaurus (Table C-2), number of index terms per search session (Table C-3), 
number of items* ** displayed (Table C-4), and number of items printed (Table C-5). 
Table 1-1 summarizes overall usage. 



*These differences are discussed in greater detail in Appendix C. 

**An ’’item" consists of a bibliographic citation and descriptive abstract 
describing a single report or document. 
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Table 1-1 

SUMMARY OF TOTAL OVERALL USAGE 



Terminal 


Operation 

Time 


Number of 
Searches 


Number of 
Items Printed 


Stanford 


7 months 


338 


30,970 


San Francisco 


7 months 


453 


20,111 


San Diego 


2 days 


40 


1,200 



As Table 1-1 indicates, in 7 months of operation nearly 1,000 searches were con- 
ducted and more than 51,000 items were printed in response to the searches. 

This volume is equivalent to that of republishing 3 years' issues of the R.I. E . 
(Research in Education) announcement catalog twice over. In this case, however, 
items were developed in response to specific inquiries. 

Table 1-2 is a summary of individual search characteristics for Stanford, San 
Francisco, and San Diego. 



Table 1-2 



SUMMARY OF AVERAGE STATISTICS PER SEARCH 



Terminal 


Elapsed 

Time 

(min) 


Thesaurus 

References 


Index 

Terms 


Items 

Displayed 


Items 

Printed 


Stanford 


23.7 


6.4 


15.0 


3.4 


91.7 


San Francisco 


19.8 


9.5 


21.9 


5.8 


44.3 


San Diego 


16.3 


8.9 


15.1 


5.9 


30.4 



The differences shown in Table 1-2 among the various locations are probably not 
particularly important. Stanford, for example, provided more in-depth instruc- 
tion and demonstration than either of the other two, which perhaps accounts for 
the longer elapsed search time. It should also be remembered that ’’search" is 
defined not as a single question but rather as a single session with a particular 
user. Frequently, several related questions are formulated during the course 
of a single search session. 



Section 2 



FINDINGS AND ANALYSIS 

This section contains evaluations of ERIC/DIALOG use at the ERIC Clearinghouse 
on Educational Media and Technology at Stanford University, and at the Region IX 
Office of Education in San Francisco. No formal evaluation was carried out for 
the San Diego Conference installation. 

Evaluation of Stanford Results * 

The ERIC Clearinghouse on Educational Media and Technology at Stanford Univer- 
sity was asked by the U. S. Office of Education to help evaluate online computer 
searching of the ERIC document file. A terminal with its supporting equipment 
was installed in the Clearinghouse, and arrangements were made for data-phone 
linkage with Lockheed’s nearby Information Sciences facilities, where the 
DIALOG system was available. 

DIALOG was to be used to accomplish Clearinghouse tasks — both those usually 
performed and those impossible without such a system — and to see whether 
individuals could sit down at a terminal and, with little preliminary instruction, 
use such a system to locate relevant educational research documents. The system 
was employed in a variety of Clearinghouse projects and proved extremely useful. 

Nine case studies were conducted, and 19 questionnaires were solicited, which 
reveal in some detail the reactions of the individuals (who had quite different back- 
grounds and sought quite different information) to 2-hour ’’hands-on” sessions at 
the terminal. 

The case studies demonstrate that individuals with no preliminary experience with 
computers or mechanized information systems can, indeed, in a reasonably short 
time, use a system such as DIALCXl effectively. 

The two aspects of the system that most impressed the nine evaluators were its 
speed and the way it ’’widened horizons,” i.e. , the way it suggested other relevant 
areas of information or different approaches to the information originally sought. 

When the evaluators were specifically asked to list ’’good” features of the system, 
its speed and its ’’horizon-widening” effect were mentioned. Also cited were the 
way in which sets of documents could be combined, using Boolean logic, and the 
fact that the system was simple to use. 

The evaluators also were specifically asked to list ”bad” features. Delays in wait- 
ing for the system to execute a command were singled out, along with the feeling 
that considerable experience or time was needed to master the operating rules. 
(While some evaluators made this last comment, others called the system simple 

*This subsection is extracted from Ref. 1 (which is recommended as a more 
complete analysis and evaluation). 
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to use and easy to work with.) Users were asked to comment on each component 
of the system, such as the typed terminal record and the cathode ray tube display. 
The terminal record was appreciated as a means of monitoring the ongoing search 
and as a way of obtaining a permanent record of search strategy. The cathode 
ray tube was cited as saving great amounts of time and as being a natural way to 
present data. There was criticism of some components, or of their integration 
in the system. A number of users wished that commands could be entered without 
having to hit so many keys. The project coordinator registered concern over 
problems with hardware and software that caused interruptions in scheduled 
service. Changes that have been made in DIALOG since the evaluation sessions 
have obviated some of the unfavorable comments. 

Seven of the nine evaluators reported that they asked additional and different 
questions about the subject they originally were investigating, and seven — 
although not the same se 3n - said tliat they came upon material on different, 
thou^ related, subjects * , at they would like to pursue at a later date. 

Verbatim statements are presented here to demonstrate user reaction to this 
aspect of the system: 

A I began to realize they had some articles of international scope . . . 

[and that] kind of opened up that area .... 

C Well, as we began to look at the section on instructional television, 
there were some related topics there that I hadn’t been aware of ... . 
There were a couple of topics that interested me, one in the area of 
teacher training, which I just happened to run across, but I would 
like to look into at another time, althou^ it wasn’t particularly 
related to this study. 

D The only thing I formulated was new areas to look under for relevant 
research. 

E Let me tell you some questions that were answered that I posed to 
myself, and were answered during the process .... New questions 
that are already answered .... I stumbled on something, again 
because there are some things that are not in the thesaurus but are 
in dialog [identifiers] . I stumbled onto [specific] programing 
languages. 

H ... The biggest problem was staying with what I had originally pur- 
sued, instead of gettir^ off on other interesting things. 

I I did run across some other information [that I would want to pursue 
later] .... We’ve got our mind on one thing, particularly when we’re 
researching something, one phase, and we only think to look in certain 
areas, and the thing that I liked about this, the machine points out 
that there may be some other related information .... This is , I 
think, very significant, and very helpful. 
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We assume that such ”horizon -widening” effects are extremely desirable, and 
we can imagine no research administrator with such a narrow area of concern 
as to object to this aspect of DIALOG. To look at the situation from a different 
viewpoint, this aspect tends to bring the useful documents in the collection to 
the user’s attention even though his preconceived ideas of what is available are 
incorrect. 

Six of the evaluators had favorable comments on the way the system made it possible 
to monitor and modify searches, but the other three should not be considered 
negative or indifferent. Most of the evaluators had no previous experience with 
computerized retrieval systems, and so could hardly compare DIALOG with non- 
interactive, batch -processing systems. Our feeling was that the evaluators 
reacted to DIALOG as an entity, and that the overwhelmingly favorable general 
comments were to a great extent the result of this very basic aspect of the system. 

The individual verbatim statements were as follows: 

A It makes a big difference, because you get a feeling of control over 
your search that you don*t have so much when you’re actually in the 
library. There, it’s hard to remember exactly which things you were 
going to go back and do — 3 mu have to write things down, you have to 
organize things. Here you have- handy little systems for putting some- 
thing off somewhere and you just organize in your own mind the very 
basic concepts. 

B It has a much more organizing effect, it helps to organize in a much 
more effective way. 

C It helped me .... I think the fundamental help was that I had some idea 
of the amount of information I was handling, or would be handling, if 
I had it printed out. And it gave me some insight, too, into the amount 
of research that had gone on in certain areas. 

D It’s like having a great mass of information at your disposal, where you 
can somehow set up and know v^ere you are and how much you’ve 
looked at. 

F I think it made a great difference. 

I It gives you other ideas and other, related, sources. 

After the nine intensive case studies had been completed, the retrieval system 
was used to conduct searches requested by visitors to the Clearinghouse. The 
observations and reactions by 19 of those visitors were recorded on six-page 
questionnaires that paralleled, to a great extent, the ’’debriefing outline” used 
in the nine case studies. By the time the first of the 19 visitors arrived on the 
scene, the previously discussed modification of programing had been completed, 
and it was DIALOG II, rather than the original system, that was observed. 



7 



o 



There were no striking differences between the responses recorded on question- 
naires by the 19 and the tape-recorded comments of the nine case study partici- 
pants. One question on the questionnaire, however, had not been put to the nine 
evaluators: 

If DIALOG were readily available to you at a convenient 

location, how often would you like to use it? 

All 18 visitors who answered this question indicated that they would like to use 
such a system several times a year or oftener. A third '^f the visitors indicated 
that they would like to use it once a week or oftener. The number giving each 
answer is sho>\’n below: 



once a 

never 

year 


several 
times 
a year 


once a 
month 


once a 
week 


more often 
than once 
a week 


0 0 


6 


6 


4 


2 



About half the visitors (8 of 18) indicated that the system revealed new aspects 
or dimensions of the subject to them, and about half (10 of 19) reported coming 
upon material on an entirely different subject that they then wished to pursue. 

The 19 ^/isitors were asked to indicate what they considered good and bad features 
of the system. A total of 65 good features and 16 bad features were listed. No 
novel comments not included in the earlier remarks of the nine evaluators were 
made. 

When prompted to evaluate specific features of the DIALOG system, the ratings 
were uniformly favorable. The ratings are presented here for the sake of com- 
pleteness. Each scale is presented as a continuum, running from the descriptive 
term presented at the left to that at the right. The number of visitors checking 
each point on the scale is indicated: 

The Keyboard 



easy difficult 

to use 6 6 4 2 0 to use 



The Typed Terminal Record 


easy 
to use 


6 


9 


3 


0 


0 


difficult 
to use 


valuable in 
conducting 


17 


0 


0 


1 


0 


not 

valuable 



a search 
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The CRT or Visual Display 



easy 

to use 10 


j6 


_0 


_1 


_g 


difficult 
to use 


valuable in 
conducting 
a search 


_0 


_g 


_0 


_o 


not 

valuable 


The Thesaurus 






easy 
to use 


_2 


_3 


__0 


_o 


difficult 
to use 


valuable in 
conducting _7 




_2 


_g 


_o 


not 

valuable 



a search 



In addition to the nine case studies presented in this report, some other observa- 
tions on DIALOG appear as a product of its application to a variety of tasks in 
the Clearinghouse, These applications can be categorized as duplicate checking, 
in-house projects, answers to information requests, and visitor use. The actual 
ejcperience gained from using DIALOG for these purposes has bearing on any 
future computerization of the ERIC files. 

Duplicate checking, for example, is an extremely time-consuming process which, 
in the past, has involved manually checking each acquisitioned document against 
the current acquisition data log and all past indexes. DIALOG has made it possible 
for the staff member in charge of this processing to perform the checking more 
quickly and with greater assurance of not missing a document. The only require- 
ment for this is that the file be as current as possible. A real improvement would 
be the ability to access the acquisition data log on the CRT, thereby streamlining 
the process still further. 

A spin off of the system is a "dump" of the indexes. This listing makes it possible 
to perform routine checking at any time, saving online processing for the especially 
difficult documents. Again, for this to be continuously effective, it would be 
necessary to keep the listing updated. This listing also has value as a I'ecord of 
the number of relevant entries in the file on any given attribute, which makes it 
possible to check the size and scope of any search before actually using the online 
terminal. 

/ 

The in-house aspect of DIALOG use is similar to visitor use in that the system can 
be used to obtain extensive bibliographies on any subject being considered for a 
locally produced bibliography or state-of-the-art paper, as well as to conduct 
literature searches to support other projects being undertaken by a clearinghouse. 
Three clearinghouses (in addition to ERIC at Stanford) have made use of this 
facility to produce long bibliographies for various projects. It has also been used 
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to produce supporting bibliographies for clearinghouse-affiliated people working 
on a project of the Institute for Educational Development, evaluating the impact 
of Title VH research.. Thus, clearinghouse activities related to anialysis of infor- 
mation in the clearinghouse *s specialty area are greatly assisted by the online 
search capability. 

The load of quite specific informational requests that flow into the Stanford Clear- 
inghouse (approximately 40 per month) can also be handled more effectively by 
using DIALOG. While it was in operation, the Clearin^ouse staff used DIALOG 
to handle 93 of these requests in depth. Without the online facility, it would have 
been impossible to give each of these users an exhaustive bibliography . Instead, 
the requests would have been handled in a general way, perhaps by giving the 
person information on where he might start his search or who might be able to 
help him. DIALOG made it possible to give the requested listings of things already 
processed by ERIC that conformed to. his particular interests. Feedback from 
people receiving this service indicates that it was extremely useful and greatly 
appreciated. An offshoot of handling these requests was a compilation of the dup- 
licates of past searches; these can be used to answer subsequent similar requests 
without using additional computer time. 

Finally, and most importantly, DIALOG proved valuable to Clearinghouse visitors. 
Through October 31, 46 people had been given a demonstration and introduction to 
the system; 68 people had their requests searched by a staff member while they 
were present to interact with the system, modifying the search strategy as neces- 
sary; and 21 people (including the nine evaluators) were taught to use the system 
and gained "hands-on” experience. Indmduals who thus used the Clearin^ouse 
as a source of ERIC materials were able to perform their literature searches 
efficiently. 

Perhaps the most important aspect of DI/lLOG is the ease of use of the system. 

For people who are unfamiliar with computers and who have a limited amount of 
time to devote to their own professional research and even less to learmng to use 
a new research technique ~ no matter how powerful it may be ~ it is essential that 
the technique be simple to understand. Experience in demonstrating the DIALOG 
system and in instructing people in its use indicates that the system is indeed 
simple and does not overwhelm a person who is unfamiliar with computers. The 
instruction did require some staff time to teach people on an individual basis, 
but it is clear that the process could be made basically self-instructional through 
development of some simple materials and a film or slide presentation. A mini- 
mal amount of staff time would be required to monitor the first use of the system 
until the user was proficient. 

Evaluation of Region IX Results (San Francisco) 

The Office of Education Region IX Office, located in San Francisco, conducted 
searches for a variety of users, including representatives from several pro- 
fessional categories and geographical regions. A report is planned by the Regional 
Office analyzing this activity; some of the statistics and questionnaires from users 
have been made available by thi.s office for inclusion in this report. 
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Professional categories of terminal users are listed in Table 2-1. 



Table 2-1 

PROFESSIONAL CATEGORIES OF TERMINAL USERS 



Category 


Percent of Total 


Teachers 


38.0 


Researchers 


24.0 


Government Personnel 


17.0 


Administrators 


10.0 


Private Citizens 


9.0 


Librarians 


2.0 


Total 


100.0 



As shown in Table 2-1, more than half of the user group were teachers or 
researchers. Within the teacher group, nearly two-thirds were elementary or 
secondary teachers, and one-third were teachers from hi^er education institu- 
tions. Government personnel included users from federal, state, and local 
education organizations. Administrators included executive director-level per- 
sonnel, irrespective of organization. Although these categories are not mutually 
exclusive, they were chosen to indicate the breadth of appeal of such retrieval. 

Table 2-2 shows the geographic distribution of system users. 

Table 2-2 

GEOGRAPHICAL DISTRIBUTION OF TERMINAL USERS 



State 


Percent of Total 


California 


89.0 


Washington 


3.0 


Kansas 


2.0 


Oregon 


2.0 


Other 


4.0 


Total 


100.0 



Clearly, most users were from California. Users in the "Other” category 
include Hawaii (3 users), Nevada (2 users), Alaska (1 user), and Washington, 

D. C. (1 user). 

In this study, 300 searches were conducted. Although most users (86%) conducted 
a single search, several (24%) conducted more than one search, as indicated in 
Table 2-3. 



Table 2-3 



TA.BUIATION OF NUMBER OF SEARCHES PER USER 



Number of Searches 
per User 


Percent of 
Total 


1 


86.0 


2 


8.0 


3 


2.0 


4 


2.0 


5-8 


2.0 


Total 


100.0 



Post-use questionnaires were sent to individuals participating in the study. The 
questionnaires surveyed a variety of user-reaction areas, including: whether the 
original search question was answered, what good features were observed, what 
poor or awkward features were observed, whether the display (as compared with 
a typewriter) was helpful, and whether or not the ERIC Thesaurus was useful. 
Table 2-4 summarizes the results of the data obtained from the questionnaire. 



Table 2-4 

SUMMARY OF QUESTIONNAIRE RESPONSES FROM OFFICE OF 

EDUCATION REGION IX OFFICE 



Questions and Replies 


Percent of 
Total Replies 


1. Were original search questions answered? 




Yes 


90.0 


No 


10.0 




100.0 


2. What good features impressed you most ? 




Speed 


30.5 


Ability to coordinate concepts 


15. 0 


Time saving over conventional methods 


13.5 


Completeness of results 


13.0 


Ability and knowledge of search coordinator 


11.0 


Interactive capability of system 


8. 5 


Convenience and simplicity of system 


8. 5 




100.0 


3. What features did you find poor or awkward ? 




None 


30.8 


Inadequate subject coverage or indexing 


18. 5 


Training required to use system 


15.4 


More time desired at terminal 


10. 5 


Time delays and system failures 


9.2 


Need for more information on document sources 


9.2 


System should be more widely available 


6.4 




100.0 
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Table 2-4 (Cent. ) 



Questions and Replies 


Percent of 
Total Replies 


4. What was your reaction to the visual display? 




It helped substantially 


97.0 


It was not too helpful 


3.0 

100.0 


5. How helpful did you find the published ERIC Thesaurus? 




Extremely helpful 


37.5 


Moderately helpful 


51.5 


Not too helpful 


11.0 

100.0 



It should be noted that replies to questions 2 and 3 were unprompted. The sum- 
mary replies listed in Table 2-4 represent a somewhat gross characterization of 
the actual comments received. Several of the actual comments are presented 
below for illustration: 

When you*re up a blind alley you can branch out without wasting time. 

[The system] could completely eliminate useless research and duplication 
of effort. (Librarian) 

Not only were original questions answered, but so were the subquestions 
which the search helped formulate. (Resource teacher) 

Our schools must chaise. The proposed changes must be built on truth. 

The retrieval of researched areas of education for change construction is 
critical. (Teacher) 

System coverage should be expanded and made generally available, as it is 
the only way ERIC materials will get wide distribution and use. (Teacher) 

Without the computer, I would have ignored ERIC as a source of data because 
of the difficulty of using R.I. E. (Teacher) 

I found dozens of papers I was previously unaware of even though I carefully 
check the monthly issues of R.I. E. (College Administrator) 

Undoubtedly ERIC would be used much more often if this system were 
generally available. (Program Director) 

It is a fantastic system; how long education has needed it! (Coordinator) 

I am so greatly impressed with the system that I can only hope that it will 
be maintained and expanded, including the availability of terminals on 
each college campus. (College Administrator) 
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Fantastic; in 22 minutes I accomplished what would have required 10 hours 
if done in a traditional mamier. (Teacher) 

As the volume of ERIC materials increases, it will be imperative that this 
system be made available to all users. (Director of Research) 

I was delighted with the results of my first search. It will save me dozens 
of hours and will allow me to produce teaching and resea7.’ch materials that 
might otherwise be impossible because of time constraints. (College teacher) 

It would have taken at least 2 months using ordinary library research methods 
to obtain such concise, pertinent information. (College President) 

To summarize the results of the Regional Office us£^e, response to the DIALOG 
online retrieval system with the ERIC files was unanimously enthusiastic. Of 
the users who indicated their search questions had been answered (90%), many 
were surprised at Ihe speed of the system and at the completeness of the results. 
Those users who indicated that their question had not been answ”ered (10%) usually 
commented that R.I. E. did not include material pertinent to their particular query. 
Speed was offered as the most favorable feature of the system, followed by the 
ability to coordinate concepts to form intersections of the data base. Negative com- 
ments referred to the data base in 28% of the cases, and to the retrieval system in 
25% of the cases. Thirty-one percent of respondents could think of no negative 
features, and 17% indicated that they desired more time on the system or that the 
system should be more wddely available. Almost all users (97%) indicated that the 
visual display helped substantially, as compared wdth a typewriter-only terminal, 
and most users (90%) found the Thesaurus at least moderately helpful. 

Comments suggest that use of ERIC materials (R.I.E. in this case) could be 
substantially increased throu^ general use of the DIALOG online retrieval system. 
During the period of this report, more than 50, OOO citations and abstracts were 
printed in response to specific requests. This volume is equivalent to the entire 
collection being republished twice. 

Two quantitative estimates of time savings were given: 10 hours, and 2 months. 
Even the lower figure results in an efifectiveness/cost ratio of more than 16:1, 
indicating that, in this case, online retrieval is not only convenient but also 
cost effective. 

Many other comments were directed at the skill and understanding of the system 
coordinator, Mrs. Katherine Alderfer. Her role in introducing the system and 
guiding the searches elicited a M^ly favorable reaction among users. 
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CONCLUSIONS 

Because of the vast amount of information available and the ever-changing needs 
of the information user, it is necessary to "package" information in a variety of 
forms and to organize it along several dimensions. Moreover, because the 
individual who is a source of information can at best only anticipate the gross, 
rather than the specific, needs of the user, and because the information user 
does not know the magnitude of the potential supply of data in the information 
collection, it is necessary to provide the user with a tool that will allow him to 
explore - and gain rapport with - the information collection. Online information 
retrieval, as characterized by the DIALOG system, provides such a tool. It 
enables the user to dynamically partition the total information collection according 
to user-specified combinations of characteristics. As the user becomes familiar 
with the potential of the file to satisfy his data needs, he can modify his specifica- 
tion to explore items in related areas of interest. 

The effectiveness of such a tool is theoretically measured in terms of the degree 
to which the tool allows the user to identify a maximum number of relevant file 
items with a minimum number of irrelevant items. 

The usefulness of this measure is somewhat limited with regard to large, 
practical (as opposed to experimental) information stores. Measurement of 
whether all relevant items have been retrieved implies exhaustive examination 
of the file. In addition, the above measure fails to consider the extent to which 
the system assists the user in formulating his query or the extent to which the 
system "widens the horizons" of the original search topic. In the final analysis, 
the effectiveness of an information retrieval system depends jointly on the use- 
fulness of the information base and the ease with which a user can obtain the 
needed information. 

The conclusions presented in the remainder of this section are organized as 
follows: 



• User satisfaction 

• Potential demand 

• Expected benefits 

Discussion within each of these topics is largely synthesized on the basis of 
previously presented material. 

User Sati sf action 

An overwhelming majority (90%) of users were satisfied with the search results 
obtained by using the DIALOG system. Of those users who indicated that they 
were not satisfied (10%), most felt that the information they were seeking was 
not in the file. Several of the unsatisfied questions pertained to statistical or 
cost studies on various special programs. 
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System components (keyboard, typed terminal record, and display) were found 
easy to moderately easy to use by 84% of the users questioned, and none found 
them difficult to use. Of those who experienced some difficulty (6%), most had 
difficulty with the keyboard device. 

A high degree of usefulness was attributed to the display (100% at Stanford and 
97% at San Francisco). In addition, 95% of Stanford users found the typed ter- 
minal record very useful (this question was not asked at San Francisco). 

The concensus in favor of the display deserves further comment. The principal 
advantage is not that of the presentation medium (i.e. , display versus typewriter), 
but rather the presentation content. Because of the slow speed of the typewriter 
(15 to 30 characters per second) compared with the display (120 to 240 characters 
per second), it is simply impossible to transfer information from the computer as 
fast as the user can assimilate it using only a typewriter device. The result is 
that most typewriter-oriented retrieval systems neither provide the user access to 
the indexing dictionary (or thesaurus), nor do they display full citations and/or 
abstracts to assist the user in better defining his search question. 

The mere use of a display in no way assures that a system provides the useful 
features discussed in the preceding paragraphs. Many typewriter-oriented 
systems merely substitute a display for the tyx)ewriter, adding no additional 
information-transfer capability. The critical questions seem to be whether the 
system provides sufficient information flow from the file to the user to enable 
him to evaluate and, if desired, to modify his search question in a truly inter- 
active mode. In this case, a simbiosis between man and machine can occur to 
the material benefit of each.* Otherwise, the only advantage of **being online” 
to a computer is that the information needs can be satisfied faster.** 

Potential Demand 



The Stanford report indicates that the Stanford Clearin^ouse receives approximately 
40 reference questions per month. Assuming 20 reference centers (19 clearinghouses 
and a central Washington information center), each of which experiences this level 
of demand, a total of approximately 10, 000 requests per year are received. This 
demand level — and it is probably understated -- could be fulfilled by one experienced 
operator using the DIALCKj system. 

A different estimate of demand was offered by the Stanford study, which indicated 
that two-thirds of the individuals questioned would use the terminal at least once 
per month. The expected level of usage based on the replies of this sample is 
25 queries per user per year. In 1968, there were more than 5, OOO subscribers to 
R.I. E. If only one system user per subscriber is assumed, this would result in 
125,000 requests, or approximately 6,250 per reference center. Such a demand 
would justify a part-time operator at each of 20 reference centers, or 12 full-time 
operators at a central center. 



*More relevant results for the user, less printing for the computer. 

**Many computer people unacquainted with the benefits of true interaction argue 
that faster response time is the only benefit of an online — as opposed to an 
offline - system. 
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Expected Benefits 

An information retrieval system coupled to an ever-expanding data base can pro- 
vide three types of services; 

• Retrospective retrieval — identification of a subset of abstracts that are 
relevant to particular detailed questions 

• Special bibliographies - production of searches of wide interest, the results 
of which are published and distributed 

• Current awareness - periodic notification provided to special-interest 
groups of current abstracts in their areas of interest 

Of the three types of services, this report considers the retrospective-retrieval use 
of the system in greatest detail. Reference 1 describes several special bibliographies* 
developed by other clearinghouses and by staff members in support of other projects. 
The current-awareness service arises as a byproduct of DIALOG. That is, any user 
requesting a search can ask that he continue to be informed of future abstracts that 
fall into his search request areas. In this case, his search expression is retained and 
processed against each update of the file to keep the user informed and up to date on 
his particular topic of interest. The result of this combination of services is that 
subscribers are continually informed on topics of interest to them with minimum incon- 
venience. Documented results of these services based on experience reported else- 
where (Ref. 2) show a substantial increase in the demand for documents announced 
within the ERIC system. One can expect increased demand in both currently and pre- 
viously announced documents. It is the expanded dissemination of documents, together 
with the greater use of educational research that expanded distribution implies, that 
indicates the direct effectiveness of the retrieval system. 

An indirect benefit to the Office of Education would accrue through use of the system 
to evaluate the novelty of requests for grants. Review of a given data base bj^ the 
grant evaluator would require that the requestor show that his investigation did not 
merley duplicate some previous research. If requestors were made aware of such a 
review procedure, they would be more inclined to use the system to review the state- 
of-the-art in their area of interest before developing a research project. ** 

Ultimately, the easy availability of an ERIC/DIALOG system allows greater normali- 
zation among members of the education profession in terms of information and access 
to research results. Consequently, the professional researcher who avails himself 
of the system is freed to pursue his own interests to a greater degree based on a 
foundation of familiarity with what others in the profession have reported. Without 
an effective information retrieval and dissemination system, the accumulation of 
knowledge within a given discipline is severely constrained. Beyond the measures 
of the abstracts printed and the documents distributed, it is undoubtedly the machinery 
for quick and efficient accumulation of knowledge that is the most significant benefit 
to be derived from the system reported herein. 



*One bibliography evaluating the impact of Title Vn research contained several 
hundred items. 

**Actual evidence of this process has been noted in the searches performed by 
some researchers. 
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N SERVICES IN THE UNIVERSITY LIBRARYt PHASE I - PL 
ANNING. PART 9. 

CALIFORNIA UNIV.t LOS ANGELES. INST. OF LIBRARY 
RESEARCH. (BBB00615) 

NATIONAL SCIENCE FOUNOATIONt WASHINGTON* D.C. ( 

FGK57295I 

DOCUMENT NOT AVAILABLE FROM EORS. 41P. 

/♦AUTOMATION/ COMPUTER PROGRAMS/ COMP(jtERS/*INFO 
RMATION CENTERS/ INFORMATION PROCESSING/* INFORMATI 
ON SERVICES/ INFORMATION STORAGE/ INFORMATION SY^ 
EMS/ LIBRARY REFERENT MAN MACMNE SYSTE 

MS/ SYSTEMS OEVELOPMENT/^ONIVERSI TY LIBRARIES 



E0023419 15 DEC 67 RIEMAR69 LI000501 NSF 6 

N-503 

EXPERIMENTAL ON-LINE MAT HEM AT ICS CITATION DSTA B 
ASE. FINAL REPORT ON MECHANIZED INFORMATION SERVIC 

JA 0F_ THE. UNIVERSITY LIBRARY* PH ASE 1-W.ANNING^ PA 



TOHPKINSi HARY 1. 

CALIFORNIA UNIV.t LOS ANGELES. INST. OF LIBRARY 

RESEARCH. (BBB006I5I 

NATIONAL SCIENCE FOUNOATIONt MASHINGTONt O.C. I 
F6K57295) 

OdCUMENT NOT' AVAUABLE' EORS; “ 3ip7 ” 

/* ABBREVIATIONS/ AUTOHA 7ION/*CI TATION INDEXES/ I 
NFORHATION PROCESSING/ INFORMATION RETR1EVAL/*INF0 
RHAfldN STORAGE/ MAN MACHINE SYSTEHS/*MATHEMATICS/ 
PERMUTED INDEXES/* SERIALS/ UNIVERSITY LIBRARIES/ 
USE STUDIES 

/MAST/ MINIMUM ABBREViATIOK OF SERIAL TITLES'TMA' 
ST I 
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60021567 25 APR 68 RIEJAN69 11000^^78 AIP/U 

DC-7„ 

AUOACiduS - AN eXPERIHENT WITH AN ON-LINEt INTER 
ACTIVE REFERENCE RETRIEVAL WENT WITH AN ON-LINEt I 
NTERACTIVE REFERENCE RETRIEVAL SYSTEM ^NG THE UN 
IVERSAL DECIMAL CLASSIFICATION AS THE INDEX LANGUA 
GE IN THE FIELD OF NUCLEAR SCIENCE. 

FREEMANf ROBERT R. ; ATHERTON, PAULINE 

AMERICAN INST. OF PHYSICSt NEW YORKt N.Y. CQPXO 
2169) 

NATIONAL SCIENCE FOUNDATION 9 WASHING! ONt D.C. f 
FGK57295) 

DOCUMENT NOT AVAILABLE FROM EORS. 3AP. 

/♦CLASSIFICATION/ DISPLAY SYSTEMS/ EVALUATION ME 
twos/ INDEXING/ INFORMATION PROCESS ING/+INFORM ATI 
ON RETRIEVAL/ INFORMATION STORAGE/«INFORMATION SYS 
TENS/ INPUT OUTPUT OEVICES/«MAN MACHINE SYSTEMS/ S 
EARCH STRATEGIES/ SYSTEMS DEVELOPMENT/ TEST RESULT 
S/ USE STUDIES 

/AUDACIOUS/ AUTOMATIC DIRECT ACCESS TO INFORMATI 
ON WITH THE ON/ NUCLEAR SCIENCE ABSTRACTS/ UDC/^UN 
IVERSAL DECIMAL CLASSIFICATION 



ED020759 APR67 CUMREPT L 1000626 

CASY^ENGLISHt A LANGUAGE FOR IJNFORJMAT ION RETRIEV 
AL THROUGH A REMOTE TYPEWRITER CONSOLE. 

RUBINOFF 9 M. ; AND OTHERS 

PiNNSYLVANJA UNlV.t PHILAOELPHI A. MOORE SCHOOL 0 
F ELECTRiCAL ENGINEERING. iSYN72350l 
DOCUMENT NOT AVAILABLE FROM EORS. 

/COMPUT ER PROGRAMS/ COMPUTERS/ INFORMATION PROCE 
SSING/^INFORMATION kETRilVAL/^iNPUt OUTPUT DEVICES 
/ MAN MACHINE SYSTEMS/ PROGRAMING 
/ EAS Y ENGLISH 



ED 02 0748 JUN67 CUHRJEPT U000417 lS R-12 

INFORMATION STORAGE AN oWfRiEVALt REPORTS ON EV 
ALUATIONt CLUSTERING, AND FEEDBACK. 

SAJLTON, GERALD 

CORNaL UNiv.f itHACA, N.Y. DEPT. OF COMPUTE SC 
lENCE. IQPX19610I 

DOCUMEN T NOT AVAI LABLE FROM EORS. 

/CLUSTER GROUPING/ COMPUTATIONAL LINGUISTICS/ CO 
RRELATION/^ EVALUATION METHODS/ INDEXING/ INFORMATI 
O N NEEDS/ INFORM ATION PRO CESS ING/4INF0RMATI0N R_E^TPL 
lEVAL/ INFORMATION STORAGE/ iNTORMAtlON SYStEf«/*I 
NFORMATION UTILIZATION/ INPUT OUTPUT/ MAN MACHINE 
SYSTEMS/ MEASUREMENT TECH NIQUES/ PERFORM^J^RiJ[E 
RIA/4SEARCH STRATEGIES/ SfATISTfCAL W fHES"^ 

AURl/ TIME SHARING/4USE STUDIES 

/ AMERICAN DOCUMENTATION I NSTITUTE/4SEARCH ^TRAT_E 
Gy7 smart I'SALfONS MAGICAL AUTOMATIC RETRIEVER OF 
TEXT) 
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ED019970 0706C66 CUMREPT L 1000357 TR-00-15 

48 

TIBRARY INFCfRM^ USING AN ON-tlNEf 



REAL-TIME COMPUTER SYSTEM, 

HOLZBAURf FREDERICK W. ; FARRISt EUGENE H, 
INTERNAtlONAL MACHINES CQRP, t POUGHKEEP 

SIEf N.Y. SYSTEMS DEVELOPMENT DIV. CQPX37995) 

EORS PRICE MF-$0,25 HC-SU96 47P, 



/AUtOHATldN/ CATALOGING/ COMPUTERS/ ELECTRONIC 0 
ATA PROCESSING/ FEEDBACK/ INFORMATION CENTERS/ INF 



ORMATION PROCESS ING/*INFORMATION SYSTEMS/ J[NP_UTJ3U 
TPUT/VINPUT output OEV library ACQUiSltldN/*L 
IBRARY TECHNICAL PRXESSES/*MAN MACHINE SYSTEMS/ T 
IME SHARING 

"/AMlNiStRAtiVE TERMINAL SYSTEM (ATS)/ IBM 1440 



COMPUTER SYSTEM 



E0019094 0CT67 CUMRB>T LI000396 NASA-CR-91 

8 

EVALUAtidN OF USER REACTIONS tO A PROTOTYPE Ofi-ll 
INE INFORMATION RETRIEVAL SYSTEM, 

MEISTERf DAVID ; SULLIVAN, DENNIS J. 

BUNKER-RAM(3 iCORP,t CANOGA PARK, CALIF, SYSTEM EF 
FECT LABORATORY, (CIQ08350) 

OOCjUMEJIT NOT AVAILABLE FROM EORS, 

/AUTOMATION/ CMPUTERS/ EVALUATION/ FEEDBACK/ IN 
FORMATION NEEOS/^INFORMAT ION RETRIEVAL/ INFORMAT 10 
„N SYSTEMS/ INTERACJION/ INTERVIEWS/ L IBRAR lANS/^MA 
FmACHTNE WSfWS/ SCiENTlFi^ PERSONNEL/ TIME SlilAR 
IN6/*USE STUDIES 

/MTIONj^AERONAUT ICS^ AND SPACE ADMINISTRATION ( 
NASA)/ QUESTlONAIRES/^ RECON 



£0019090 14SEP67 CUMREPt L 1000032 

GRINS, AN ON-LINE STRUCTURE FOR THE NEGOTIATION 
OJF^ INQUIRIES,, STUDIES IN THE MAN-SYSTEM INTERFACE 
“In irBRARIES/^R 

GREEN, JAMES SPROAT 

LEHIGH UNIV, , BETHLEHEM, PA, CENTER ’FOR THE INFO 
RMATION SCIENCES. (SYN42375) 

DOCUMENT NOT AVAILABLE FROM EORS, 

/COMPUTER PROGRAMS/ DECISION MAKING/ FEEDBACK/ I 
NblXiNG/ INFORMATION CENTERS/ INFORMATION PROCESSI 
NG/4INF0RMAT10N RETRIEVAL/ INTER AC TI ON/*MAN MACHIN 
E SYSTEMS/ PROBLEM SOLVING/ QUESTION ANSWER INTERV 
lEWS/ QUESTIONING TECHN IQUES/^SEARCH STRATEGIES/ T 
IME SHARING 

/GE-225 COMPUTER/ GRINS (GENERAL RETRIEVAL INQUI 
RY NEGOTIATION STRUCTURE) 
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E0017294 0EC67 CUMilEPT LI000209 

SPIRES (STANFORD PHYSICS INFORMATION RETRI EViiL S 
YSTEM), ANNUAL REPORT. 

PARKER f EDWIN B. 

STANFORD UNlV.t CALIF. IMST. FOR COMMUNITY RESEA 
RCH. ICIQ82350J 

DOCUMENT NOT AVAILABLE FROM EDRS. 

/COMPUTERS/ DISPLAY SYSTEMS/ FEEDBACK/* INFORMATl 
ON NEEDS/ INFORMATION PROCESSING/*INFORM ATION RETR 
lEVAL/ INFORMATION SYSTEMS/* INFORMAT ION UTILIZATIO 
N/*MAN MACHINE SYSTEMS/ NUCLEAR PHYSICS/ RESEARCH 
UTILIZATION 

/♦MAN MACHINE SYSTEM/ STANFORD PHYSICS INFORMATI 
ON RETRIEVAL SYSTEM fSP IRES)/ STANFORD UNIVERSITIE 
S LIBRARY 



EDO 1 72 83 6SEP66 CUMREPT LI 0001 01 

ON-LINE INTERACTIVE DISPLAYS IN APPLICATION TO L 
INOUmi^CL . an AL YS J.S AN 0 INF OR MAT I QN PROCE SSI N G.. AN 0. _ 
RETRIEVAL.. 

SIMMONSt R«F. 

SYSTEM DEVELOPMENT CORP.t SANTA MONICA, CALIF. 
(CIQ85350) 

DOCUMENT NOT AVAILABLE FROM EORS • 
/CQMPUTATlONAL„LlNGmSTICS/ DECISION MAKI NG / *D1S 
PLAY SYSTEMS/ FEEDBACK/* INFOR NATION PROCESSING/ IN 
FORMATION RETRIEVAL/ KERNEL SENT ENC6S/*MAN MACHINE 
SYSTEMS/ SENTENCE STRUCTURE/ SYNTAX/ TIME MARING. 
/BIBLIOGRAPHIC ON LINE DISPLAY (BOLD)/ PATTERN L 
EARNING PARSER (PLP 2) 



ED016499 04AU666 CUMREPT L 1000087 AO-661-6 

571 SP-2557 

INTERACTIVE DISPLAYS FOR DOCUMENT RETRIEVAL. 
BORKO, H. ; BURNAUGHf H.P. 

SYSTEM DEVELOPMENT CORP.t SANTA MONICA, CALIF. 

(cio'esasoj 

DOCUMENT NOT AVAILABLE FROM EORS. 

/AUTOMATION/ CLASS IF ICATION/*D IS PLAY SYSTEMS/ IN 
OEXES (LOCATERS)/*INFORMATION PROC£SSING/*INFORMAT 
ION RETRIEVAL/ INFORMATION SYSTEMS/*TIME SHARING 
/BIBLIOGRAPHIC ON-LINE DISPLAY/ BOLD/ CLASSIFICA 
TiON ^iNFbRMATiON PROCESSING/ *01 SPLAY UNITS (COMPU 
TERS)/ INDEXES (LOCATORS)/ TELETYPEWRITER 
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Appendix A 

ERIC/DIALOG USERS MANUAL 



USERS MANUAL 
ERIC/DIALOG ONLINE 
RETRIEVAL SYSTEM 



Stanford University 



N-6R-69-2 



May 1969 



Developed by: 

Information Sciences Laboratory 
Lockheed Palo Alto Research Laboratory 
LOCKHEED MISSILES & SPACE COMPANY' 

A Group Division of Lockheed Aircraft Corporation 
Palo Alto, California 94304 



FOREWORD 



Project DIALOG is the culmination of 6 years of research in information storage and 
retrieval conducted by the Information Sciences groiQ) of the Lockheed Palo Alto 
Research Laboratory. The DIALOG system offers the user an interactive command 
language with which he can direct the data processing power of a computer to assist 
him in the retrieval of information. The current version has evolved from the opera- 
tional experience of several data bases, and has proved both easy to operate and 
powerful in results . 



By eliminating intermediaries and placing the user in direct communication with the i 

computer, results are available in seconds or minutes instead of days or weeks, and 
interesting avenues of ejqploration suggested by preliminary results can be pursued 
without time lapse. DIALOG commands provide means for describing a search in- ^ 

terest to the computer, for obtaining displays (and printouts) of results, and for : 

redefining a search interest based on examination of intermediate results. By 
dynamically partitioning the information store, the user can rapidly and efficiently 
converge on a relevant set of documents. I 



In the current application, the DIALOG retrieval language is applied to the ERIC data 
base of some 15, 000 report citations. A remote terminal has been installed in the 
ERIC Clearinghouse at Stanford University, W. J. Paisley, Director. Questions and/or 
comments regarding this project should be forwarded to J. J. Prevel, U.S. Office of 
Education, Department of Health, Education, and Welfare, Washington, D. C., 

Code 20202. Additional information regarding the DIALOG system can be obtained 
from R. K. Summit of the Lockheed Palo Alto Research Laboratory. 
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Section 1 

ERIC/DIALOG DESCRIPTION 



»IALOG is an interactive information retrieval language which allows the user to 
)rmulate simple or complex search requests via a video/keyboard display terminal 
Pig. 1) wtiich is coupled to a computer containing the material to be searched. To 
se DIALOG, the user enters desired commands such as ’’begin search,” ’’display,” 
select,” or ’’combine” by depressing keys on the keyboard. The computer responds 
y displaying various data on the display screen. 

’our impoirtant phases in the search are as follows: 

• Select index terms or subject phrases 

• Combine individual terms or phrases 

• Display output 

• IVEodify search ejq)ression 

1 the first step, the user identifies and selects index terms (i. e. , subject phrases) of 
iterest wliich relate to his search topic. The second step allows the user to build his 
wn specialized subject heading during the search. At each step the user is informed 
f the numtoer of documents which fulfill his specified search criteria. The third step, 
isplay of the output, allows the user to review' references for relevancy, and to select 
dditional terms to better express his search requirement. Once the desired literature 
eferences are displayed, the user can press the ’’print” key for a permanent record 
f the retrieved references, or can modify his search eiqpression based on index terms 
sed in displayed references. 

'o use the system, a searcher must learn the clerical procedure, and must learn to 
ffectively utilize the functions performed by the commands. The remainder of this 
ection describes the clerical procedure for command entry, and provides an annotated 
earch example illustrating the commands and the associated computer responses, 
ections 2 and 3 present detailed command descriptions and ERIC/DIALOG descriptor 
odes together with examples. 
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Fig. 1 Video Keyboard Display Terminal 





LOCKHEED PALO ALTO RESEARCH LABORATORY 

LOCKHEED MISSILES I. SPACE COMPANY 
A GROUP DIVISION Of LOCKHEED AIRCRAFT CORPORATION 


















COMMAND ENTRY 



The user communicates every command by performing the following three steps: 

1. Depress desired command key (while holding down the ''SHIFT'' key). The 
command S 3 nnbol v/ill immediately appear on the display screen. 

2. Key in appropriate operand data . These data will appear on the screen 
following the command symbol. (Note: up to this point, an erroneous entry 
may be corrected by backspacing and typing over undesired data. ) 

3- Depress the ENTER key (while holding down "SHIFT"). This action signals 
the computer that the user has a message to transmit. 

The computer reacts by reading and processing the command message. A reply will 
always return on console printer (which serves as a search log) and, depending on the 
command, on the display screen as well. 



To select the index term (IT) "computers," for example, the following operations 
would be performed: 

Step Operation Keyboard (See Fig. 2) 



1 

2 

3 



Depress (with SHIFT key): 
Type in 

Depress (with SHIFT key): 



SELECT 

term 



IT/COMPUTERS 



ENTER 



This entry sequence causes the following response to be typed out on the console 
printer (Fig. 1): 






SET NO. IN DESCRIPTION OF SET f 

NO. SET (+ = OR, * = AND, - = NOT) 



1 130 IT/COMPUTERS 



In other words, the 130 items indexed by the index term "computer" are assigned a set 
number (i.e. , "1") which can be used for referencing this group of items in later commands. 



tNote: This heading information results from the BEGIN SEARCH command which is 
described in detail in Section 2. 
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ANNOTATED SEARCH EXAMPLE 




Assume that the user is interested in "use of computers in information retrieval." 

His search strategy could be as shown in Fig. 3. The user first selects the index 
term "computers." Noting that 130 items are indexed by this term, he suspects that 
"information processing" may also have been used as an index term. Selecting this 
term creates Set 2 which contains 104 items. Because he is interested in either 
"computer" or "information processing" (as they apply to information retrieval), he 
logically sums these two sets into Set 3 with the command COMBINE 1+2 (where 1 
and 2 refer to Sets 1 and 2, respectively, and + stands for UH). This set contains 
213 items. 

The user is interested in only those items of Set 3 which deal with information re- 
trieval, so he selects the index term "information retrieval" which creates Set 4 with 
119 items. To obtain the intersection of Sets 3 and 4 (those items referring to com- 
puters or information processing as well as information retrieval), he issues the 
command COMBINE 3*4 (where * stands for AND). The references in Set 5 are 
then displayed one at a time by successively depressing the ENTER key. (The eighth 
item of the set is illustrated. ) The abstract for any displayed citation can be dis- 
played by depressing DISPLAY and ENTER. 

The reader should mentally practice these steps using Fig. 2 before attempting to do 
his own search to assure an understanding of clerical and logical procedures ^ a 
search. 
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’!’y])owrilcr Hcs|Hmsc‘ 



Depress 
Key below: 
(SHUT) 



I H’r 

iJ'rm 



SET NO. IN DESCRIPTION OF SET 

NO. SET (+=OR, * = AND,-=NOT) 



Type in: IT/COMPUTERS 



Depress: 

(SHIFT) 



KNrKM 



♦ 



1 130 IT/COMPUTERS 



Depress 
Key below: 
(SHIFT) 



SEI.KCT 

term 



Type in: ITANFORMATION PROCESSING 



Depress 
Key below: 
(SfflFT) 



rOMRINE 
Mt (iMCrlptlOfl 



Type in: 



U2 



3 



104 IT/INFORMATION PROCESSING 



213 1+2 



Depress 
Key below: 
(SHIFT) 



•ELECT 

term 



Type in: ITANFORMATION RETRIEVAL 



4 119 ITANFORMATION RETRIEVAL 



Depress 
Key below: 
(SraPT) 



COMBINE 
•«C description 



Type in: 



3*4 



5 47 (l+2)*4 





Depress 
Key below: 
(SmFT) 



Mt/fmt/ltrm 



Type in: 



5 



DHTLAT l/H/l 

noiMN oenf uMMM i(AaA-ai<eM 

tVAiMTioH or tnra ntAcnoM 10 A pfMTonrFt ott'Ltiii mvottuTtoN MranvAL mr 



MMrtn. DAVID eULUVAIt. OCNIMi. 

OQMNeWMKCt-IUllOOODr.. CAMOOArAM. CAUf. 
tin^ sw Att moii tSM. 

/••tvonuTioN DtmtvAiyniAit macmm nwnm/nm mom/ autoiiation/ ooMront 
M/ (VAUiAtiON/ rtnuACK/ toroMMATioH Httm/ wroolUTiOM mrcMW otrsMcnoN/ 1 
NrcmrinM/ umumamw MATfONALAcnoiuoTioi Aiio»ACg AOMtnrnuTKm (naiai/ out 
mmmta/ lucoM/tcitimficpCMomftiy timcmamm 

KHTm tlXT CCVMAMls 





*ENTER must be depressed with each command entry to transmit the command message to the computer. 
This (^ration will not be shown with succeeding commands for the sake of brevity. 



Fig. 3 Illustrative Search Procedure 
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SEARCH EXTENSION 



Figure 4 contains the console typewriter record of an extension of the search just 
described. The first column contains a command "echo" (one per line) of commands 
entered (S- for SELECT. C for COMBINE, etc. , as in Fig. 2), and associated operand 
data (e.g. , IT/COMPUTERS). 



SEARCH TITLE; USE OF COMPUTERS IN INFORMATION RETRIEVAL 
DATE; 02/26/69 

REQUESTOR; J. J. PREVEL, USOE 

SET NO. IN DESCRIPTION OF SET 

COM^LAND-OPERAND(S) NO. SET (+ = OR, * = AND, - = NOT) 



S-IT/COMF LITERS 1 

E-IT/INFORIVL-\TION PR 
S-E5 2 

C-1-2 3 

S-IT/INFORMATION RE 4 

C-3*4 5 

D-5 

S-IT/TIME SHARIiNG 6 

C-5*6 7 

D-7 
P-7 



T-7/1 

ED016414 ED016499 ED017283 



1-30 IT/COMPUTERS 

104 IT/INFORMATION PROCESSING 
2? 3 1+2 

113 IT/INFORMATION RETRIEVAL 
47 (1+2) *4 

16 IT/TIME SHARING 
6 (1+2) *4 *6 



1-6 ITEMS HAVE BEEN PRINTED. 

TYPE 7/1/1 -6 
ED079090 EDO19094 EDO20748 



Fig. 4 Typewriter Record Including Search Extension 

It can be seen that after viewing some of the 47 references of Set 5, the user decided 
to restrict the search to those dealing with "time sharing." Selecting this term re- 
sulted in Set 6 containing 16 items. This set was then combined with Set 5 to produce 
6 items. The user displayed a few of these items and, determining them to be relevant, 
caused them to be printed at the computer with a PRINT 7 command. The accessiorx 
numbers were also desired. These were typed at the terminal by a TYPE 7/1 com- 
mand (1 indicates an "accession numbers only" format). This search illustrates a 
single procedure for identifying the relevant subset of items from a reference corpus. 

In actuality, there are several additional features available which are described in 
Section 2. Also, there is a variety of descriptor types which may be used in parallel 
fashion to IT/-index term. These include AU /-author, PS/ -primary source, 
CN/-contract number, etc. , which are described in Section 3. 



A-8 



LOCKHEED PALO ALTO RESEARCH LABORATORY 

LOCKHEED MISSILES 4 SPACE COMPANY 
A CPOUr DIVISION OF LOCKHEED AIRCKAFT CORPORATION 



SUMMARY OF DISPLAYABLE COMMAND RESULTS 



Two commands, EXPAND and DISPLAY, result in displays. These commands are 
illustrated in summary. Operand data are shown in quotation marks and it is pre- 
sumed that ENTER is depressed at the conclusion of command entry (as discussed 
on page 3). 

Command message: EXPAND ’TT/INFORMATION PROCESSING” 



Result: 

V y 



EXPAND-rr/LNFORMATION PROCESSING 



REF 


DESCRIPTOR 


CITATIONS REL. 


TERMS REF 


El 


IT/INFORMATION CENTERS 


47 


£1 


E2 


rr/lNFORMATION DISSEMINATION 


218 


E2 


E3 


rr/LSFORMATlON DYNAMICS CORPORATION 


1 


E3 


E4 


rr.'lNrORMATION NEEDS 


34 


E4 


ES 


•ttAnformation processing 


104 


E3 


E6 


itanformation retrieval 


119 


£6 


E7 


rr/INFORMA*nON science 


17 


E7 


E8 


rr/INFORMAllON SEEKING 


4 


£8 


E9 


rr/INFORMATION SERVICES 


34 


E9 



ENTER NEXT COMMAND^ 



This command is used to view alphabetically near index terms (to the entered term) 
which appear in the index term dictionary. Note that the entered term in this case is 
given a reference number of E5. Such displayed terms can be selected by using their 
associated ”E" number (as opposed to re-keying the term). 
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Command message; DISPLAY ’'5/2/8" 





Result: 



DISPLAY 5/2/8 

EDO19094 OCT67 LIOOOJ96 NASACU-918 

EVALUATION OF USER REACTIONS TO A PROTOTYI>E ON-LINE INFORMATION RETRIEVAL SYST 
EM. 

MEISTER, DAVID SULLIVAN, DENNIS J. 

CIQ08350BUNKER-HAMO CORP., CANOGA PARK, CALIF. 

DOCUMENT NOT AVAILABLE FROM EDRS. 

/•INFORMATION RETRIEVAL/*MAN MACHINE SYS'rEMS/*USE STUDIES/ A ITOMATION/ COMPUTE 
RS/ EVALUATION/ FEEDBACK/ INFORMATION NEEDS/ INFORMATION SYS^tEMS/ INTERACTION/ I 
NTERVIEWS/ UBRARIANS/ NATIONAL AERONAUTICS AND SPACE ADMINISTRATION (NASA)/ QUE 
STIONAIRES/ RECON/ SCIENTIFIC PERSONNEL/ TIME SHARING 
ENTER NEXT COMMAND^ 





This command is used to view citations from sets (resulting from SELECT or COMBINE 
commands). The above result indicates the citation is from Set 5, is displayed in 
format 2, and is the eighth item in the set. 

Abstracts for any displayed citation can be displayed by simply depressing the DISPLAY 
key (and the ENTER key). The following display shows the first page of the abstract 
corresponding to the first illustrated citation. 




DISPLAY 5/4/8 

68A19094 

REACTIONS OF A USER POPULATION (SaENTlSTS/ENGINEERS AND UERARUNS) TO AN OPER 
ATIONAL TEST OF AN ON-LINE SYSTEM FOR RETRIEVING DOCUMENT CITATIONS (RECON) UTRE 
STUDIFD .AT SEVERAL NASA FACILmES* DURING A TWO MONTH OPERATIONAL TEST PERIOD. 
DATA WERE COLLECTTED REGARDING FREQUENCY OF USAGE AND DEGREE OF SUCCESS ENCOUNTE 
RED IN CONDUCTING ON-LINE SEARCHES. ADDITIONAL DATA WERE GATHERED BY OBSERVATION 
, INTERVIEWS. QUESTIONNAIRES AND CONTROLLED EXPERIMENTAL TESTS CONCERNING THOSE 
FEATURES OF THE SYSTEM WHICH WERE MOST ACCEPTABLE TO THE USER POPULATION AND THE 
ADEQUACY WTTH WHICH THIS SYSTEM COULD RETRIEVE CITATIONS LN COMPARISON WTTH OFF 
-UNE MACHINE SEARCH METHODS. THE USER POPULATION USED THE SYSTEM EXTENSIVELY 
ENTER NEXT COMMAND^ -MORE • 
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Section 2 

COMMAND DESCRIPTION 



• BEGIN SEARCH 

Depressing this key (together with the SHIFT key) will cause a display on the screen 
requesting the date, the user^s name, and the title of the search. The user will 
type this information in the area provided, depressing the ENTER (shift) key on 
completion of each item. 

• EXPAND term (see page 7 for example) 

Depression of EXPAND and entry of a term cause a display of descriptors that are 
alphabetically close to the entered term . With each descriptor is shown the number 
of citations in which the descriptor appears as well as the number of conceptually 
related terms which are available as descriptors. Each display descriptor is 
numbered (El through E9) and can be referenced by this number (while being dis- 
played) in other EXPAND or SELECT commands. The number of related terms 
(if any) for any displayed term are shown in the right-hand column of the display. 
These conceptually related terms may be displayed by depressing EXPAND with 
the desired E number reference. 

• SELECT term (see page 6 for example) 

Terms may be selected either by referencing the E number of an EXPAND display 
or by entering the term itself. SELECT adds the term so entered to the user’s 
search descriptor list, and makes the set of citations associated with the term 
more readily available to the computer. A set identification number is assigned 
by the computer for use in subsequent COMBINE and output commands. 

• COMBINE set description (see page 6 for example) 

Depression of this command key, together with entry of a set description, causes 
the generation of a new set corresponding to the operation specified in the set 
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description. A set description is a series of set numbers separated by one of the 
following operator symbols: 

Symbol Operation 

+ OR 

* AND 

NOT 

The set description 1+2+3 means that any citation containing index term 1 or term 
2 or term 3 will be returned. This operator symbol is used to broaden the scope 
of a search by groiq)ing similar terms such as ’’defect or flaw or porosity.” The 
set description 1*2*3 means that any citation retrieved must contain all of the 
terms 1 and 2 and 3. This operator is used to narrow the scope of a search by 
requiring the common occurrence of several index terms such as ’’WELD and 
DEFECT and ALUMINUM.” By using the results of one COMBINE command in the 
definition of another, sets such as (l+2)*(3+4) can be developed. An example of 
such a set could be that containing (WELD or ARC WELD) and (DEFECT or FLAW). 
In this case, returned items will contain WELD and DEFECT, WELD and FLAW, 
ARC WELD and DEFECT, or ARC WELD and FLAW. At the conclusion of each 
combine command, the computer assigns a set number to the results, indicates the 
number of entries in the set, and develops an e^anded set description in terms of 
individual subject term set numbers . 

Special note: A consecutive string of sets appearing on the console printer may be 
combined by depressing COMBINE, keying in the range of set numbers to be com- 
bined followed by a slash, and keying in the desired operator (+ for ”or,” * for 
’’and”) (e.g. , sets 1 through 5 can be OR’d by COMBINE 1-5/+). 

• LIMIT set/yr/type/range 

This command, although generally not applicable to the ERIC file, can be used to 
limit sets to certain specified accession number ranges. To limit Set 5 to the range 
ED010000-ED015000, for example, it is necessary to enter: 

LIMIT ’’5/ALL/ALL/10000-15000” 
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If it is desired to limit all subsequent sets in a similar manner, it is necessary to 
enter: 



LIMIT "ALL/ALL/ALL/iOOOO-15000" 



• KEEP set/item 

set is the set identification number appearing on the console typewrHer 

item may either be a single number (e. g. , 3), a range of numbers (e.g. , 5-9), or 
the word ALL indicating which items in a set are to be kept. If "item" is not speci- 
fied, ALL is assumed by the computer. To keep an item currently being displayed, 
it is merely necessary to depress KEEP (and ENTER). 

The results of this command are placed in a set arbitrarily numbered 99. Set 99 
may be displayed, printed, or combined with other sets. 

• DISPLAY set/fmt/item 

• TYPE set/fmt/item 

, • PRINT set/fmt/item 

set is a set identification number which appears on the console typewriter 

fmt is any one of three format codes requested: e.g. , 1 (accession numbers); 

2 (formatted citations); 3 (full citation, unformatted); 4 (abstract); 5 (formatted 
citation and abstract). 

item may either be a single number (e. g. , 3), a range of numbers (e.g. , 5-9), or 
the word ALL indicating which items in a citation set are to be output. If "item" 
is not specified, ALL is assumed by the computer. 



DISPLAY (see page 7 for example): The citation(s) requested are displayed on the 
video console by depressing this command key, entering the desired set/fmt/item. 



and depressing ENTER. If a range of citations is requested, successive citations are 



displayed by depressing ENTER to change the display. The abstract for any displayed 
citation (format 2) can be displayed by depressing DISPLAY (and ENTER). This 
operation does not interrupt the normal sequencing of a previous DISPLAY command. 

TYPE: The citations indicated are typed out on the typewriter whenever this command 
key is depressed. This command should not be depressed until the search is complete. 

h 

i PRINT: Citations are printed off-line and delivered to the Clearinghouse the next day. 
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TYPE or PRINT (and ENTER) can be used to output any displayed citation immediately. 
A display sequence is not interrupted by this operation. 



Special note: If format 2 and all items of a set are desired in an output command 
(DISPLAY, TYPE, PRINT) it is necessary only to enter the command and the set 
number (e.g. , DISPLAY 5 is equivalent to DISPLAY 5/2/ALL). 



• END SEARCH 

This command terminates the search, and requests comments from the user. 
Depression of the final ENTER key (after comments have been entered) clears 
the computer for the next search. 
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Section 3 

ERIC/DIALOG DESCRIPTOR CODES AND EXAMPLES 



This section presents descriptor codes and examples. The citation display for 
ED013371 is shown. This is the same citation shown and described in the front matter 
of recent issues of Research in Education (Fig. 5). The descriptor code examples 
are taken from this citation. 




DISPLAY ED013371/2 

ED013371 15APR66 64 AA000223 NDEA-7B-449 OEC-4-16-023 

ICONIC SIGNS AND SYMBOLS IN AUDIOVISUAL COMMUNICATION. AN ANALYTICAL SURVEY OF 
SELECTED WRITINGS AND RESEARCH FINDINGS. HNAL REPORT. 

NORBERG. KENNETH D. 

CIQ77250SACR*\MENTO STATE COLL. , CAUF. 

EDRS PRICE MF-IO.75 HC-$5.24 131P. 

/•BIBUOGRAPHIES/*COMMUNICATION (THOUGHT TRANSFER)/*PERCEPTION/»PICTORIAL STIM 
UU/*SYMBOLIC LANGUAGE/ INSTRUCTIONAL TECHNOLOGY/ VISUAL STIMUU 



V J 



Code 


Example 


Description 


AC/ 


AC/64 


Area Code - Code for 0. E. legislative pro- 
gram which supported research activity 


AU/ 


AU/NORBERG, KENNETH D. 


Author of document 


CN/ 


CN/OEC-4-16-023 


Contract Number 


DN/ 


DN/NDEA-VnB-449 


Document Number - Assigned by originator 
of document 


IT/ 


IT/BIBUOGRAPHIES (etc.) 


Index Terms - Subject terms assigned by an 
indexer to characterize doc-jment contents 


PS/ 


PS/SACRAMENTO STATE 
COLL., CALIF. 


Primary Source ~ Organization at which 
document originated 


SC/ 


SC/CIQ77250 


Source Code — Code number for originating 
organization 


TI/ 


TI/ICON IC SIGNS AND 
SYMBOLS. . . 


Title “ The first 47 characters of the title 


YR/ 


YR/66 


Year - Publication year of document 



These descriptor type codes are included in an index listing of the ERIC/DIALOG 
retrieval index which is available at the terminal. 
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ERIC Accession Number— identifica- 
tion number sequentially assigned to 
documents as they are processed. 




Authorfs). 



Title. 

Organization where document origin- 
ate! 



Date published.- 



Contract or Grant Number--contract 
numbers have OEC prefixes; grant 
numbers have OEG prefixes. 




Alternate source for obtaining docu- 
ments. 



EDRS Price— price through ERIC 
Document Reproduction Service. 
**MF’ means microfiche; "HC” 
means hard copy. When listed “not 
available from EDRS” other sources 
are cited above. 





ED 013 371 64 ' — AA 000 223 

Korberg, Kenneth D. 

ICONIC SIGNS AND SYMBOLS IN 
AUDIOVISUAL CO.MMUNICATION, 
AN ANALYTICAL SURVEY OF SE- 
LECTED WRITINGS AND RF^EARCH 
FINDINGS, HNAL REPORT. 

Sacramento State CoU., Calif. 

Spons Agency — USOE Bur of Research 

Report No.— NDEA-VIIB~449 

Pub Date — IS Apr 66 
Contract— OEC-4* 1 6-023 
Note — Speech given before the 22nd Na- 
tional Conference on Higher Education, 
Chicago, 111., 7 Mar 66. 

Available from — Indiana University Press, 
lOtb and Morton St., Bloomington, In- 
diana 47401 ($2.95) 

EDRS Price— MF-$0.75 HC-$5.24 129p. 
Descriptors — ^Bibliographies, *Communi- 
cation (thought transfer), *Perception, 
^Pictorial Stimuli, *SymboIic Language, In- 
siructicnal Technology, Visual Stimuli. 
Identifiers — Stanford Binet Test, V^echsler 
Intelligence Scale; Lisp 1.5; Cupertino < 
Union School District 
The field of analogic, or iconic, signs was 
explored to (I) develop an annotated bibli- 
ography and (2) prepare an analysis of the 
subject area. The scope of the study was 
limited to only those com^nents of mes- 
sages, instructional materials, and com- 
municative stimuli that can be described . 
properly as iconic. The author based the 
study on a definition of an iconic sign as 
one' that looks like the thing it represents. 
The bibliography was intended to be repre- 
sentative and reasonably comprehensive 
and to give emphasis to current rese.arch. 
The analysis explored the nature of ic onic 
signs as reflected in the literauie and re- 
search. The conclusion of the analysis at- 
tempted to relate some issues in perception 
theory to the problem iZ «he development 
of a theory of iconic '!gns. Discussions 
were intiuded on (1) the stimulus-response 
paradigm, (2) the psychophysical theory of 
perception, (3) an information theory ap- 
proach, (4) nonveibal communication and 
pictic analysis, (5> a theory of pictorial 
communication and (6) perception and non- 
linear signs. (AL) — 




Legislative Authority (!ode (or iden- 
tifying the legislation which sup- 
ported the research activity (when 
applicable). 

Clearinghouse accession number. 

Sponsoring Agency— agency respon- 
sible for initiating, funding, and 
managing the research project 

Report Number and/or Bureau Num- 
ber-assigned by originator. 



Descriptive Note. 

Descriptors— subject terms which 
c'aracterize substantive contend. 
Only the major terms, preceded by 
an asterisk, are printed in the sidi- 
ject index. 



Identifiers— additional idenfi^ni 

terms not found in the Thasaums M 
ERIC Descriptors. 

Informative Abstract 



Abstractor’s initials. 



Fig. 5 Sample Entry (Extracted From Research in Education , Feb 1969) 
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Appendix B 

EXAMPLES OF INVITATION LETTERS SENT 
OUT FROM SEARCH CENTERS 



B-1 




STANFORD UNIVERSITY 

STANFORD. CALIFORNIA 94305 



INSTITUTE FOR COMMUNICATION RESEARCH 
ERIC Clearinghouse on Educational Media and Technology 



CYPRESS HALL 
(415) 321.2300 
Extension 30S1 



March 14, 1969 



Dear Colleague in Education: 

You are invited to see "ERIC On Line," at the Stanford ERIC 
Clearinghouse. Thanks to an advanced computer system developed by the 
Lockheed Missiles and Space Company, the citations and abstracts of 
more than 15,090 documents in the ERIC system can be searched in 
seconds, viewed on a cathode-ray display console, and printed for 
permanent reference. 

The system permits fast, simple interaction between user and 
computer. The user types in index terms, authors, or titles of interest 
to him. He may use many terms 'in combination (e.g., "televised instruc- 
tion" or "educational television" and "disadvantaged youth" or 
"disadvantaged groups"); this is a powerful information retrieval strategy 
that is not possible in printed, noncomputerized indexes. When the 
user has Inspected the documents that meet his search request, he can 
select and combine documents into a "relevant set" for printing. If the 
search has not identified as many documents as he needs or expects to 
see, he can expand his request, view more citations on the cathode-ray 
display, and enlarge his "relevant set." The system will also display 
lists of authors with similar names or lists of similar index terms, 
if the user needs such assistance in forming his search request. 

Educators and researchers who have already visited our 
Clearinghouse to see "ERIC On Line" are very pleased with its performance. 
They share our hope that a system like this will eventually connect local 
information centers, and even individual schools, i;' a national on-line 
educational information system. 

At the present time, "ERIC On Line" operates Tuesdays, 

Wednesdays and Fridays. We have demonstration hours scheduled from 1 to 
2 PM on those days. The system will be operating at least until the end 
of the summer. 

If you can, please stop by and see us. Bring your own search 
request so that we can find out how well the system serves your needs. 

It would help us, in scheduling our own use of the computer, to know 
when you are coming. 




I 
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William Paisley 
Dlrectoxr 
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OfFiCE OF EDUCATION 



DEPARTMENT OF HEALTH. EDUCATION. AND WELFARE 

REGIONAL OFriCE 

760 MARKET STREET. ROOM 837 
SAN FRANCISCO. CALIFORNIA 94102 

December 2 $, I 969 



Dear Colleague in Qiucation: 

On August 21, 1969 ^ we sent letters to almost two hundred educational 
administrators and decision makers, inviting them to observe and to use 
the eric/dialog ON-LINE RESPRIEVAL SYSTEM in our Begional Office q.uarters 
in Sen Francisco* Most of the addressees were located in the metropolitan 
San Francisco area. 

Our August letter explained that "ERIC On-Line" was an esqperiment scheduled 
for public use from September 2 throu^ October 31, I 969 . We are now 
happy to azinounce that this experimental operation has been extended 
throu^ July 15 , 1970, by decision of the Office of InformatlOKi Dissemina- 
tion, U. S« Office of Education, Washington, D. C. The USOE decision was 
pronpted in large part by the extensive and generally very favorable 
response of the many educators who were able to take advantage of our first 
invitation. 

At this time, we are not only Treating our invitation for the benefit of 
those indiv iduals who could not be accommodated by the previous ti^t 
scheduling, but we are also further encouraging the use of "ERIC On-Line" 
by educational researchers. The latter group had access to an "ERIC 
On-Line" facility which had been operated by the ERIC Clearin^ouse on 
Educational Media and Technology, at Stanford University# Thus, for the 
time being, we are the only USOE Regional Office carrying on the further 
es^eriment^L use of an "ERIC On-Line" retrieval system. 

Althou^ it is Just possible that two additional hours per week for using 
"ERIC On-Line" may be available, our firm ^.^^.odule is; Tuesdays) 

Wednesdays and Fridays, from 1:00 p.m. to j; .) p#m. 

If you can, please drop in to see the "ERIC On-Line" system in operation. 
Better yet, please bring your own search request so that we can continue 
to analyze how well the system serves your needs,; To e3q>edite your search, 
pj^ease phone us in advance (Mrs. Katherine Alderfer, at 415-556-8986) so 
that we can schedule your visit within the limited time available. A 
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member of our staff vill be on duty to assist you in every possible vty 
in your dialog with "ERIC On-Line." And to make our service even more 
efficient, we ask that you take a minute or two nw to CGoplate the 
attached one-page "Pre-Dialog" questionnaire and present it to us when 
you come to our office for a specific information search* 

In conclusion, a brief description of "ERIC On-Line** is still considered 
appropriate, even though it was described in our letter of August 21, 1969 • 
The "ERIC On-Line" is an advanced conputer system developed by Lockheed 
Missiles and ^ace Conpany. Through this system, the citations and 
abstracts of approximately 25,000 documents in the ERIC collection of 
educational research documents can be searched in seconds, viewed on a 
cathode-ray display console, end printed for permanent reference* 

The ^stem permits fast, slnple interaction between user and conputer* 

She user types the index terms, authors, or titles of Interest to hla. 

He may use many terms in combination (e*g*, "televised Instruction or 
"educational television" and "disadvantaged youth" "disadvantaged 
grocps") • Biis is a powerful information retrieval strategy that is not - 
possible in printed, nonconputerized indexes* Wien the user has injected 
the iocumeats that meet his search request, he can select and combine 
docTiments into a "relevant set" for printing* If the search has not 
identified as many documents as he needs or expects to see, he can expand 
his request, view more citations on the cathode-ray di^lay, and enlarge 
his "relevant set." The system will also display lists of authors with 
similar names or lists of similar index terms, if the user needs such 
assistance in forming his search request. 



Sincerely yours. 




Walter Hirsch 

Director, Bdueational Research 
Region 9 



Attachment 



ERIC PRE-DIALOG QUESTIOimiRE - Region 9^ USOE 



Address: 

Telephone: 

Agency or “bistituliion: 

Your title or job: 

Date: 



1« Please describe briefly the infonnatlon you vlU be looking for in the ERIC files: 



2 * About how many documents do you e:q)ect to find in the ERIC file on *0113 topic? 

(PLEASE CIRCLE OWE) 

Less than 10 10-25 26-50 51-100 More than 100 

3 . Have you ever used the ERIC publication, RESEARCH IW EDUCATION? YES NO 

4. (if Yes:) Have you searched RIE by subject? YES HO 

Have you searched RIE by title? YES NO 

Have you searched RIE by author? YES NO 

Have you cliosen any terms from the ERIC THESAURUS to begin your search? YES NO 
(if YES:) Please list the principal ones: 

6* How familiar are you vith on-line retrieval systems? (PLEASE CHECK EACH ANSWER 
THAT APPLIES) 

( ) l*veheard about them 
( ) I've read about them 
( ) I've studied them in some detail 

( ) I've designed or cooperated in the design of an on-line system 

( ) None of the above 

7 • How much experience with DIALOG have you had? (PLEASE CHECK EACH ANSWER THAT APPLIES) 

( ) I have previously used DIALOG myself, sitting at the console 

( ) I have watched another person use DIALOG 

( ) I have had no previous experience with DIAIXXJ 

8. How much experience have you had with on-line retrieval systems other than DIAIOG7 
(PLEASE CHECK THE ANSV7ER THAT BEST APPLIES) 

( ) I have used several systems, each on mote than one occasion 

( ) I have used several systems, each on one occasion 

( ) I have used one system on more than one occasion 

I have used one system on one occasion 
I have watched others using system(s) 

I have had no previous esqperience with on-line systems 
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Appendix C 

DETAILED SUMMARY OF SEARCH RESULTS AND 
COMPARATIVE EVALUATION OF EQUIPMENT 



Appendix C provides frequency distributions summarizing system use. The dis- 
tributions are recorded for Stanford, San Francisco, and a conference at San 
Diego. Also included in this appendix is a sample listing of search titles to 
acquaint the reader with the variety of questions asked of the system. Finally, 
a detailed description and comparison of the different equipment used is presented. 

Usage Statics 

Usage statistics are gathered automatically by the DIALOG system. Tabulation, 
however, is manual. To obtain the data shown in the tables in this appendix, sam- 
ples of 100 search histories were taken from Stanford and from San Francisco. A 
sample of 25 searches was used from the San Diego conference. 

Usage is described according to elapsed time per search session (which may 
include several search questions), online references to the Thesaurus, number 
of index terms selected per search session, number of concept groups per search 
expression, number of clerical errors per session, number of abstracts displayed, 
and number of abstracts printed. 

Table C-1 shows the distributions of elapsed search time per search session. It 
should be realized that a search session frequently included several (usually re- 
lated) search questions. Furthermore, sessions, were often used for instruction 
of unfamiliar users . 

The high average time at Stanford resulted from a high incidence of demonstration 
and instruction sessions. The low average time at San Diego reflects an attempt 
to demonstrate the system to as many people as possible consistent with producing 
a reasonably inclusive bibliography. 

Table C-2 indicates the number of Thesaurus references made by computer. The 
Thesaurus is referenced to assist the user in finding descriptors (index terms) that 
are appropriate to a particular search question. Generally, the more familiar one 
is with the collection, the less frequently this command is used. Some users pre- 
reference the printed ERIC Thesaurus, which also reduces the necessity for online 
use. 

Few, if any, retrieval systems (other than DIALOG) provide online access to the 
Thesaurus. As indicated in Table C-2, this feature is heavily used, when available. 



Table C-1 



ELAPSED SEARCH TIME 



Search Time 
(min) 


Percent of Searches 


Stanford 


San Francisco 


San Diego 


0-10 


28.0 


21.0 


8.0 


11-20 


24.0 


45.0 


36.0 


21 -30 


16.0 


17.0 


32.0 


31 -40 


12.0 


11.0 


12.0 


41 -50 


8.0 


2.0 


8.0 


51-60 


4.0 


1.0 


4.0 


61-70 


8.0 


3.0 


0.0 


Average Time 








per Search 


23.7 


19.8 


16.3 



Table C-2 

NUMBER OF THESAURUS REFERENCES PER 
SEARCH SESSION 



Number of 
References 
per Session 


Percent of Searches 


Stanford 


San Francisco 


San Diego 


0 


25.0 


10.0 


0.0 


1 -5 


39.0 


22., 0 


20.0 


6-10 


19.0 


29.0 


56.0 


11-15 


12.0 


24.0 


12,0 


16-20 


6.0 


6.0 


8.0 


21 -25 


2.0 


5.0 


4.0 


26 -30 


1.0 


3.0 


0.0 


31 -35 


1.0 


0.0 


0.0 


36 -40 


0.0 


1.0 


0.0 


Average 

Thesaurus 

References 


6.4 


9.5 


8.9 
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Table C-3 depicts the number of terms per search session. This measure indi- 
cates the number of different descriptors selected by the user in the course of 
developing his search question(s). This measure is indicative of the power of 
a coordinate retrieval system in that it is generally not feasible to accomplish 
unions and intersections of sets of abstracts in manual systems. 

Table C-3 

NUMBER OF INDEX TERMS SELECTED PER SEARCH SESSION 



Number of Index 
Terms Selected 


Percent of Searches 


Stanford 


San Francisco 


San Diego 


1- 


-10 


46.0 


29.0 


36.0 


11- 


-20 


35.0 


40.0 


40.0 


21- 


-30 


8.0 


9.0 


16.0 


31- 


-40 


8.0 


9.0 


8.0 


41- 


-50 


1.0 


5.0 


0.0 


51- 


-60 


0.0 


3.0 


0.0 


61- 


-70 


0.0 


2.0 


0.0 


71 - 


-80 


1.0 


0.0 


0.0 


81- 


-90 


0.0 


2.0 


0.0 


91- 


-100 


1.0 


1.0 


0.0 


Average IndbA 
Terms per Search 


15.0 


21.9 


15.1 



Table C-4 shows a distribution of abstracts displayed per search. The display 
of abstracts allows the user to evaluate the relevance of items retrieved for 
particular search expressions. Normally, the more items displayed, the 
higher the relevance of the final output, and the fewer the number of items 
printed. 



Table C-4 

NUMBER OF ABSTRACTS DISPLAYED PER SEARCH SESSION 



Number of Abstracts 


Percent of Searches 


Displayed 


Stanford 


San Francisco 


San Diego 


0 


57.0 


1(5.0 


8.0 


1-5 


21.0 


46.0 


32.0 


6-10 


10.0 


21.0 


36.0 


11 - 15 


6.0 


10.0 


16.0 


16-20 


3.0 


6.0 




21-25 


3.0 


1.0 




Average Number of 
Abstracts Displayed 


3.4 


5.8 


5.9 



Table C-5 is a distribution of the number of items printed per search session. 
The printed output usually consists of a printout of both bibliographic citations 
(title, author, subject items, etc.) and descriptive abstracts. This is the final 
product of the search and represents a special bibliography developed at the 
remote terminal to answer a particular question. 

Table C-5 

NUMBER OF CITATIONS AND ABSTRACTS PRINTED 
PER SEARCH SESSION 



Number of Citations 
and Abstracts Printed 


Percent of Searches 


Stanford 


San Francisco 


San Diego 


0 


9 


0 


8 


1-20 


31 


34 


28 


21-40 


11 


25 


32 


41-60 


7 


22 


28 


61-80 


8 


7 


1 


81 - 100 


6 


4 


1 


101 - 120 


4 


2 


1 


121 - 140 


2 


0 


1 


141 - 160 


4 


2 




161 - 180 


2 


1 




181-200 


1 


0 




Over 200 


5 


3 




Average Number 
of Items Printed 


91.7 


44.3 


30.4 



Sample Search Topics 

The following topics are examples of the kinds of inquiries that may be addressed 
to the system: 

Instructional terms 
Student activism 
Research in literature 
Educational parks and laboratories 
Research and Development Centers 
Drug abuse 

Innovations in curricula 

High school guidance 

Ecology and conservation education 

Vocational counseling and career planning 

Optic chiasma 

Supervision of student teachers 
Changing attitudes toward race problems 
Discrimination toward females 
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student attitudes toward teacher behavior 
Creative programs for infants 
Black studies 
Diffusion of innovations 
History of educational technology 
Instructional materials for handicapped children 
Multi -cultural education 
Individual instruction projects 
Evaluation of educational media 
Social studies and the disadvantaged 
Utilization of educational technology 
Production of instructional materials 
Generally useful skills for job placement 
Use of computers in higher education 
^ Suicide among college students 
Large-group instruction 
Programed instruction for the deaf 
Selection of library materials 
Use of simulation and gaming techniques 
Cost effectiveness in school administration 

Evaluation of Equipment 

The Stanford facility used an IBM 2260 terminal that operated at 2400 bits/sec 
(240 characters/sec) over a Bell Systems 201B data set. Both the San Francisco 
and Washington facilities used a Computer Communications , Inc . (CCI) CC30 
terminal. Table C-6 provides a comparison of pertinent terminal characteristics. 

Table C-6 

COMPARISON OF TERMINAL CHARACTERISTICS AND COSTS 



Item 


Characteristics and Costs 


IBM 2260 


CC30 


Screen Size (Characters) 
Transmission Speed 
Telephone Modems 
ModeiSfRental per Month 
Terminal Rental per Month 


960 ch. 

240 ch./sec 
201B 
$85.00 
$705.00 


960 ch. 

120 ch./sec 
202D 
$45.00 
$323. 00 



In addition to the above characteristics, the IBM terminal requires 220- V current, 
air conditioning, and a dedicated, leased telephone line. There was slightly 
higher reliability with this configuration, however, than with the CCI terminal, 
which utilizes ordinary dial-up telephone lines (n.b., the CC30 terminal can be 
configured with the more reliable telephone lines). The latter terminal uses 
standard 100-V current, requires no air conditioning, and is portable. 



User reaction to both terminals was uniformly favorable with regard to display 
format, ease of viewing, and keyboard operation. 

An experimental evaluation was conducted utilising a computer-controlled micro- 
filin display device manufactured by MicroForm Data ^sterns. In this mode of 
operation, the user communicates his search strategy to the computer through 
an ordinary teletype or typewriter terminal. Instead of the computer returring 
the full, text of the retrieved abstracts to the terminal, it merely sends back 
addresses of retrieved abstracts to the microfilm display device. Each of the 
addresses positions selected abstracts, contained on microfilm, for display on 
the microfilm display. 

Althou^ the microfilm device was operational by the end of the experimental period, 
the images were of less than satisfactory viewing quality, and the positioning 
mechanism was somevhat unreliable. It is felt that image quality could be improved 
somewhat by producing the films directly from the abstract computer tapes (rather 
than optically refilming the images as was actually done). This configuration was 
demonstrated at the 1969 Fall Joint Computer Conference at Las Vegas. 



C-6 




